classdef dotsQueryable < dotsAllManagedObjects
    % @class dotsQueryable
    % Superclass for objects that acquire data and classify it.
    % @details
    % The dotsQueryable superclass provides a uniform way to manage data,
    % such as data from a gamepad or eyetracker.  It specifies how to
    % format raw data and imposes a model for treating data samples as
    % points in space.  It also provides a fast way to classify data, by
    % associating regions in data space with arbitrary outputs.
    % @details
    % <b>Spatial Model</b>
    % @details
    % It's useful to model data as regions in space.  For example, imagine
    % a gamepad with three elements: x-axis, y-axis, and a button. The
    % three elements imply a three dimensional space, with samples from
    % each element falling along one of the spatial dimensions.  A point in
    % space corresponds to one data sample form each element.  Likewise, a
    % rectangular box in space corresponds to a range of neighboring values
    % from each element.
    % @details
    % dotsQueryable uses the dotsFiniteDimension and dotsPhenomenon classes
    % to represent data space and regions in space.  These can represent
    % spaces of any dimensionality (within the limits of Matlab).
    % @details
    % dotsFiniteDimension and dotsPhenomenon require that the data space be
    % finite and discrete.  This makes the spatial model computationally
    % manageable.  As a result, the spatial model only treats samples
    % within a specified range of interest and at a finite granularity.
    % Note that that this constraint does not apply to raw data stored a
    % dotsQueryable object.  Raw data may use the full range and precision
    % of Matlab's double numbers.
    % @details
    % <b>Classification</b>
    % @details
    % The spatial model allows for fast classification of data.  The most
    % recent data samples, one from each element, indicate a point in the
    % data space.  dotsQueryable can classify reagions in space by
    % associating them with with meaningful outputs.  Since the data space
    % is finite and discrete, dotsQueryable can convert raw data to
    % a meaningful output only using arithmetic and array indexing, and
    % none of the loops that slow down Matlab!
    % @details
    % This classification combines well with topsStateMachine.  For
    % example, imagine one dotsQueryable object and one topsStateMachine
    % object:
    %   - the queryable object may configure its classifications so
    %   that the outputs match the names of states in the state machine.
    %   - the state machine may use the queryAsOfTime() method of the
    %   queryable object as the input for its states.
    %   .
    % With this combination, the states machine's state traversal path will
    % depend on data classification performed by the queryable object.
    % @details
    % <b>State, not Signals</b>
    % @details
    % dotsQueryable focuses on classifying the @e state of a data set at a
    % particular moment in time and can capture interactions among data
    % dimensions at that time.  It does not attempt to do @e signal @e
    % detection for any data dimension over time.
    % @details
    % However, dotsQueryable may track selected dimensions over
    % time and automatically save a queue of timestamps from when the
    % dimensions had value changes.  The queryNextTracked() method reads
    % through the timestamp queue and returns the data classification
    % corresponding to that point in time.  This may help to make sure that
    % no data of interest slip by unnoticed and unclassified.
    % @details
    % <b>Subclasses</b>
    % @details
    % dotsQueryable itself is not a usable class, it just provides a
    % uniform interface and core functionality to subclasses (see below).
    % Subclasses must fill in some details:
    %   - checkIsAvailable() This is a mandatory method which subclasses
    %   must redefine.  It should return true if the object is ready to be
    %   used, or else false.  This is a good place to do any initial setup
    %   or hardware configuration.
    %   - setupDimensions() This is a mandatory method which subclasses
    %   must redefine.  It must fill in the dimensions property with an
    %   array of dotsFiniteDimension objects.  It may also fill in the
    %   isTrackingDimension property with an array of true/false, one for
    %   each dimension.
    %   - setupDefaultPhenomenons() This is an optional method which
    %   subclasses may redefine.  It should create dotsPhenomenon objects
    %   to represent regions of data space that are likeley to be resued,
    %   such the region corresponding to a button press.  It save the
    %   phenomenon objects using the addPhenomenonToGroup() method.
    %   - readData() This is a mandatory method which subclasses must
    %   redefine.  It must get new data, for example by querying a gampad
    %   device.  It must save the data using the appendData() method.  See
    %   the allData property for the correct data format.
    %   - appendData() This is an optional method which subclasses may
    %   redefine.  Normally the appendData() method defined in
    %   dotsQueryable is sufficient.  But some subclasses may with to
    %   preprocess all incoming data, in which case they should extend
    %   allData().
    %   - flushData() This is an optional method which subclasses may
    %   redefine.  Normally the flushData() method defined in dotsQueryable
    %   is sufficient.  But some subclasses might need to do things like
    %   reset hardware, flush a queue, etc., at the same time that data are
    %   flushed, in which case they should extend flushData().
    %   .
    % @ingroup queryable
    
    properties
        % true or false, whether the object is ready to be used.
        % Subclasses can determine whether they're available, for example
        % when a cartain device is present, by defining the
        % checkIsAvailable() method.  checkIsAvailable() is called when the
        % object is initialized and the result is saved in isAvailable.
        % @details
        % Automatically opts in to share via dotsTheSwitchboard.
        isAvailable = false;
        
        % array of dotsFiniteDimension objects.
        % Subclasses must define the setupDimensions() method which must
        % add an array of dotsFiniteDimension to the dimenions property.
        % The array of dimensions defines and describes the data space used
        % by the object.  Each dimension may leave its indexes property
        % empty but must supply values for its other properties.
        dimensions = [];
        
        % number of data dimensions in the data space, same as the length
        % of the dimensions property.
        nDimensions = 0;
        
        % array of true/false, whether to track changes on each dimension.
        % Where isTrackingDimension is true, appendData() will add
        % timestamps to the list of timestamps in trackingTimestamps, for
        % later classification by queryNextTracked().  Default is all
        % false--don't track any dimensions.
        % @details
        % Tracking dimensions alloes dotsQueryable subclassses to take
        % advantage of event notification and avoid polling for data.
        isTrackingDimension = false(0,0);
        
        % array of timestamps for value changes on tracked dimensions.
        % See isTrackingDimension and queryNextTracked().
        % @details
        % flushData() clears all the timestamps saved trackingTimestamps.
        trackingTimestamps;
        
        % nx3 double matrix of acquired, formatted data.
        % Subclasses must add data to the allData matrix in a particular
        % format:
        %   - each row represents a data sample
        %   - each row must have the format [ID, value, timestamp]
        %   - the first column must contain ID integers indicating to which
        %   data dimension each sample belongs.  Each ID must match the ID
        %   property of one of the dotsFiniteDimension objects in
        %   dimensions.
        %   - the second column must contain the raw data samples.
        %   - the third column must contain a timestamp for each sample.
        % @details
        % The number of rows of allData, n, may grow with each call to
        % appendData().
        allData = zeros(0,3);
        
        % mx3 double matrix of the most recent data.
        % currentState contains the most recent sample from each data
        % dimension.  The number of rows of currentState, m, is the same as
        % nDimensions. Each row has the same format as the rows in allData.
        % @details
        % Before data are aqquired, the second column of currentState will
        % contain the neutralValue for each dimension.  The third column
        % will contain -inf instead of timestamps.
        currentState = zeros(0,3);
        
        % the smallest timestamp found in allData.
        earliestDataTime = -inf;
        
        % the largest timestamp found in allData.
        latestDataTime = inf;
        
        % topsGroupedList containing groups of dotsPhenomenon objects.  The
        % list groups are arbitrary.  The list mnemonics correspond to the
        % name properties of the phenomenon objects.
        % @details
        % The phenomenon objects should represent regions of interest in
        % the object's data space.  These might be supplied when the object
        % is initialized, by the setupDefaultPhenomenons() method, or at
        % any time afterwards for custom behavior, using the
        % addPhenomenonToGroup() method.
        phenomenons;
        
        % topsGroupedList containing pairs of dotsPhenomenon objects and
        % arbitrary outputs.  The list groups are arbitrary.  The list
        % mnemonics are numeric rankings for sorting classifications by
        % importance.
        % @details
        % Each classifications assign an arbitrary, meaningful value to a
        % region of the object's device space.  Where regions in space
        % overlap, the meaning with the highest ranking takes precedence.
        % By default, no classifications exist.  They may be added with the
        % addClassificationInGroupWithRank() method.
        % @details
        % One group of of classifications may be activated at a time by
        % setting the value of activeClassificationGroup.  As
        % activeClassificationGroup changes, the meanings assigned to
        % regions in space may change.
        % @details
        % The queryAsOfTime() method uses data from a moment in time to
        % locate a region in the object's data space.  If a classification
        % is activated for that region, queryAsOfTime() returns the output
        % associated with that region.
        classifications;
        
        % one of the groups from the classifications list.
        % Setting the value of activeClassificationGroup changes the
        % meanings assigned to regions in the object's data space.
        activeClassificationGroup;
        
        % containers.Map of outputs from the queryAsOfTime() method.
        % queryOutputs uses timestamps as keys to store the outputs from
        % the queryAsOfTime() method.  Thus, it keeps a history of queries.
        % @details
        % flushData() clears all the keys and outputs saved in
        % queryOutputs.
        queryOutputs;
        
        % cell array of possible classifications outputs, sorted by ranking
        activeOutputs;
        
        % the output from the most recent queryAsOfTime().
        % latestQueryOutput stores the output from the most recent call to
        % queryAsOfTime(), or [] if queryAsOfTime has not yet produced an
        % output.
        % @details
        % flushData() sets latestQueryOutput to [].
        latestQueryOutput = [];
        
        % placeholder output for queryAsOfTime() or queryNextTracked().
        % Query methods return the unavailableOutput when the object is
        % unavailable.  This allows for custom, non-trivial fallback
        % behavior.
        unavailableOutput = 'unavailable';
        
        % placeholder output for queryNextTracked().
        % queryNextTracked() returns the upToDateOutput when there are no
        % more value changed on tracked changes waiting to be queried.
        % This allows for custom, non-trivial fallback behavior.
        upToDateOutput = 'up to date';
    end
    
    properties (Hidden)
        % output lookup table with same size as phenomenon space
        activeLookupTable;
        
        % numeric class of integers to use for lookup tables
        lookupTableIntClass = 'uint16';
        
        % dimensions index lookup table by integer IDs
        dimensionByID;
        
        % scale factors for decimating raw data into data space subscripts
        dataGranularities;
        
        % coefficients for data space subscripts to linear array indexes
        dataSubscriptCoefficients;
    end
    
    methods
        % Constructor takes no arguments.
        function self = dotsQueryable
            self = self@dotsAllManagedObjects;
            self.initialize;
        end
        
        % Reinitialize the object.
        function initialize(self)
            self.phenomenons = topsGroupedList;
            self.classifications = topsGroupedList;
            
            self.queryOutputs = ...
                containers.Map(-1, 'a', 'uniformValues', false);
            self.queryOutputs.remove(self.queryOutputs.keys);
            
            % subclass must redefine
            self.isAvailable = self.checkIsAvailable;
            if self.isAvailable
                self.initializeDimensions;
            else
                disp(sprintf('%s is unavailable', class(self)));
            end
        end
        
        % Reinitialize dimensions and phenomenons.
        function initializeDimensions(self)
            self.dimensions = dotsFiniteDimension.empty;
            self.isTrackingDimension = false(0,0);
            self.setupDimensions;
            self.nDimensions = length(self.dimensions);
            self.prepareDataLookup;
            self.setupDefaultPhenomenons;
            self.flushData;
        end
        
        % Determine whether the object is ready to use.
        function isAvailable = checkIsAvailable(self)
            isAvailable = false;
        end
        
        % Clear out acquired data and queryAsOfTime() outputs.
        function flushData(self)
            if self.isAvailable
                self.currentState = self.getNeutralState;
                self.allData = zeros(0,3);
                self.earliestDataTime = inf;
                self.latestDataTime = -inf;
                self.queryOutputs.remove(self.queryOutputs.keys);
                self.latestQueryOutput = [];
                self.trackingTimestamps = [];
            end
        end
        
        % Add allData to the topsDataLog.
        % @details
        % Adds the current allData matrix to topsDataLog, using the group
        % name "@e class.allData", where @e class is the object's class
        % name.
        function logAllData(self)
            groupName = sprintf('%s.allData', class(self));
            topsDataLog.logDataInGroup(self.allData, groupName);
        end

        % Add each of queryOutputs to the topsDataLog.
        % @details
        % Adds each value stored in queryOutputs to topsDataLog, using the
        % query timestamp the data timestamp and the group name
        % "@e class.output", where @e class is this object's class name.
        function logQueryOutputs(self)
            groupName = sprintf('%s.output', class(self));
            keys = self.queryOutputs.keys;
            if ~isempty(keys)
                timestamps = [keys{:}];
                for t = timestamps
                    data = self.queryOutputs(t);
                    topsDataLog.logDataInGroup(data, groupName, t);
                end
            end
        end
        
        % Get new data, format it, and pass it to appendData.
        function readData(self)
            newDataTime = nan;
        end
        
        % Add formatted data to allData and do accounting.
        % @param data nx3 double matrix of formatted data.
        % @details
        % See allData for the correct data format.
        % @details
        % Where isTrackingDimension is true, new data for the corresponding
        % dimensions will be tracked: the data timestamps will be appended
        % to trackingTimestamps.  Use queryNextTracked() to query data for
        % each tracked timestamp.
        function appendData(self, data)
            % save all data
            self.allData = cat(1, self.allData, data);
            
            % summarize most recent data
            %   - assume data for each dimension sorted by increasing time
            %   - assume Matlab reassigns in order encountered
            dims = self.dimensionByID(data(:,1));
            self.currentState(dims,:) = data;
            
            % account for data times
            self.earliestDataTime = min(self.allData(:,3));
            self.latestDataTime = max(self.allData(:,3));
            
            % append times for tracked dimensions
            dimIsTracked = self.isTrackingDimension(dims);
            trackTimes = data(dimIsTracked,3);
            nTrackTimes = numel(trackTimes);
            self.trackingTimestamps(end+1:end+nTrackTimes) = trackTimes;
        end
        
        % Classify data from the given time and get the associated output.
        % @param time classify data with timestamps less than or equal to
        % @a time.  Optional, defaults to the latest time.
        % @details
        % Uses data values from @a time to identify a region in the data
        % space and returns the output associated with that region.  If no
        % output is associated with the region, returns [].
        % @details
        % If the object is not ready to use (isAvailable is false), returns
        % unavailableOutput.  This may allow for non-trivial fallback
        % behaviors when, for example, a hardware device is missing.
        function output = queryAsOfTime(self, time)
            if self.isAvailable
                
                if nargin < 2
                    state = self.currentState;
                else
                    state = self.getStateAtTime(time);
                end
                
                % round state values to granularity
                %   by converting to phenomenon space subscripts
                dimValues = state(:,2)';
                dimMins = [self.dimensions.minValue];
                dimSubscripts = round( ...
                    (dimValues-dimMins)./self.dataGranularities);
                
                % clip subscripts to phenomenon space size
                paddedSubscripts = zeros(2, self.nDimensions);
                paddedSubscripts(1,:) = dimSubscripts;
                paddedSubscripts(2,:) = 0;
                clippedSubscripts = max(paddedSubscripts, [], 1);
                
                dimSizes = [self.dimensions.size];
                paddedSubscripts(1,:) = clippedSubscripts;
                paddedSubscripts(2,:) = dimSizes-1;
                clippedSubscripts = min(paddedSubscripts, [], 1);
                
                grandIndex = 1 + ...
                    sum(clippedSubscripts.*self.dataSubscriptCoefficients);
                
                outputIndex = self.activeLookupTable(grandIndex);
                if outputIndex > 0
                    output = self.activeOutputs{outputIndex};
                    self.queryOutputs(max(state(:,3))) = output;
                    self.latestQueryOutput = output;
                else
                    output = [];
                end
                
            else
                output = self.unavailableOutput;
            end
        end
        
        % Classify data from the next queued tracking time.
        % @details
        % Picks the earliest timestamp from trackingTimestamps and uses
        % queryAsOfTime() to classify the data from that time.  Discards
        % the timestamp.
        % @details
        % If there are no timestamps trackingTimestamps, returns
        % upToDateOutput to indicate that tracking us up to date.
        function output = queryNextTracked(self)
            if self.isAvailable
                
                trackTimes = self.trackingTimestamps;
                if isempty(trackTimes)
                    output = self.upToDateOutput;
                    
                else
                    % timestamps should be sorted
                    %   ignore redundant timestamps
                    time = trackTimes(1);
                    redundant = trackTimes == time;
                    self.trackingTimestamps = trackTimes(~redundant);
                    output = self.queryAsOfTime(time);
                end
            else
                output = self.unavailableOutput;
            end
        end
        
        % Get the default, least-informative-possible state of the object.
        function state = getNeutralState(self)
            state = zeros(self.nDimensions, 3);
            state(:,1) = [self.dimensions.ID];
            state(:,2) = [self.dimensions.neutralValue];
            state(:,3) = -inf;
        end
        
        % Get the state of the object from an arbitrary moment in time.
        function state = getStateAtTime(self, time)
            state = self.getNeutralState;
            timeSelect = self.allData(:,3) <= time;
            if any(timeSelect)
                thenData = self.allData(timeSelect, :);
                dims = self.dimensionByID(thenData(:,1));
                state(dims,:) = thenData;
            end
        end
        
        % Get a dotsPhenomenon object to use as a template.
        % @details
        % Returns a phenomenon object that uses the same data space as this
        % object.
        function phenom = getPhenomenonTemplate(self)
            name = sprintf('%s-template', class(self));
            phenom = dotsPhenomenon.template(name, self.dimensions);
        end
        
        % Save a dotsPhenomenon object for future use
        % @param phenom a dotsPhenomenon object that partitions this
        % object's data space somehow.
        % @param an arbitrary string to group @a phenom with related
        % phenomenon objects
        % @details
        % @a phenom will be added to the object's phenomenons property,
        % which is a topsGroupedList, using @a group as the list group and
        % the name property of @a phenom as the list mnemonic.
        function addPhenomenonToGroup(self, phenom, group)
            self.phenomenons.addItemToGroupWithMnemonic( ...
                phenom, group, phenom.name);
        end
        
        % Get all the dotsPhenomenon objects from a group.
        % @param group the string identifying a group of related phenomenon
        % objects
        % @details
        % Returns a cell array of dotsPhenomenon objects from @a group.
        % Also returns, as a second output, a cell array of the names of
        % the phenomenon objects.
        function [phenoms, names] = getAllPhenomenonsFromGroup(self, group)
            [phenoms, names] = self.phenomenons.getAllItemsFromGroup(group);
        end
        
        % Associate a region in data space with an arbitrary output.
        % @param phenom a dotsPhenomenon object that represents a region of
        % interest in this object's data space
        % @param output any variable to associate with @a phenom
        % @param group an arbitrary string to group @a phenom and @a output
        % with related classifications
        % @param rank a numeric rank used to sort classifications with
        % @a group
        % @details
        % A classification pairs a dotsPhenomenon object, which represents
        % a region of interest in data space, with an output, which may be
        % anything.
        % @details
        % Classifications are stored groups, with one group active at a
        % given time.  Setting the value activeClassificationGroup
        % activates the group with that name.  Within groups,
        % classifications are sorted by rank so that overlapping regions of
        % interest may resolve to the single, highest-ranking output.
        % @details
        % The queryAsOfTime() method uses data from a given moment in time
        % to locate a region in data space and return the outuput
        % associated with that region.
        function addClassificationInGroupWithRank(self, ...
                phenom, output, group, rank)
            c.phenom = phenom;
            c.output = output;
            self.classifications.addItemToGroupWithMnemonic(c, group, rank);
        end
        
        % Build a data space lookup table for the new group.
        function set.activeClassificationGroup(self, group)
            self.activeClassificationGroup = group;
            
            % expect group items to be sorted by increasing rank
            %   thus, highest rank is added to the table last, "on top"
            [cs, ranks] = self.classifications.getAllItemsFromGroup(group);
            self.activeOutputs = cell(size(cs));
            dimSizes = [self.dimensions.size];
            self.activeLookupTable = zeros(dimSizes, self.lookupTableIntClass);
            for ii = 1:length(cs)
                c = cs{ii};
                self.activeOutputs{ii} = c.output;
                self.activeLookupTable(c.phenom.space) = ii;
            end
        end
    end
    
    methods (Access = protected)
        
        % Define the data space for this object.
        function setupDimensions(self)
            self.dimensions = dotsFiniteDimension.empty;
        end
        
        % Identify handy, reusable regions in the object's data space.
        function setupDefaultPhenomenons(self)
        end
        
        % Do accounting for data dimenions.
        function prepareDataLookup(self)
            
            % lookup table to get dimensions array index from integer ID
            self.dimensionByID = ...
                zeros(1, max([self.dimensions.ID]), self.lookupTableIntClass);
            
            % scale factor to convert real values to dimension subscripts
            self.dataGranularities = zeros(1, self.nDimensions);
            for ii = 1:self.nDimensions
                d = self.dimensions(ii);
                
                self.dimensionByID(d.ID) = ii;
                if d.size <= 1
                    self.dataGranularities(ii) = inf;
                else
                    self.dataGranularities(ii) = ...
                        (d.maxValue - d.minValue) / (d.size - 1);
                end
            end
            
            self.currentState = self.getNeutralState;
            
            % coefficients used to compute grand linear index from
            % individial dimension subscripts
            self.dataSubscriptCoefficients = ...
                cumprod([1, self.dimensions(1:end-1).size]);
            
            % if subclass did not supply isTrackingDimension
            %   fill in default--all false
            if isempty(self.isTrackingDimension)
                self.isTrackingDimension = false(1, self.nDimensions);
            end
            
            dimSizes = [self.dimensions.size];
            self.activeLookupTable = zeros(dimSizes, self.lookupTableIntClass);
        end
        
        % @todo building a lookup table on demand turns out to be slow.  Maybe
        % build lookup tables as they change and cache them, along with their
        % outputs cell array, in a map.  Then, for example, queryAsOfTime()
        % could take a string and use the corresponding table and outputs.
        %   - should make it fast to queryAsOfTime under different
        %   conditions
        %   - how much space would multiple tables take up?
    end
end